SPIN-PM: A consensus framework to evaluate the presence of spin in studies on prediction models.

OBJECTIVES: To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model, regardless of the modelling technique. STUDY DESIGN: We followed a three-phase consensus process: (1) pre-meeting literature review to generate items to be included; (2) a series of structured meetings to provide comments, discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) post-meeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed. RESULTS: This consensus process involved a panel of eight researchers and resulted in SPIN-PM which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples. CONCLUSION: We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

RELEASE-HF study: a protocol for an observational, registry-based study on the effectiveness of telemedicine in heart failure in the Netherlands.

INTRODUCTION: Meta-analyses show postive effects of telemedicine in heart failure (HF) management on hospitalisation, mortality and costs. However, these effects are heterogeneous due to variation in the included HF population, the telemedicine components and the quality of the comparator usual care. Still, telemedicine is gaining acceptance in HF management. The current nationwide study aims to identify (1) in which subgroup(s) of patients with HF telemedicine is (cost-)effective and (2) which components of telemedicine are most (cost-)effective. METHODS AND ANALYSIS: The RELEASE-HF ('REsponsible roLl-out of E-heAlth through Systematic Evaluation - Heart Failure') study is a multicentre, observational, registry-based cohort study that plans to enrol 6480 patients with HF using data from the HF registry facilitated by the Netherlands Heart Registration. Collected data include patient characteristics, treatment information and clinical outcomes, and are measured at HF diagnosis and at 6 and 12 months afterwards. The components of telemedicine are described at the hospital level based on closed-ended interviews with clinicians and at the patient level based on additional data extracted from electronic health records and telemedicine-generated data. The costs of telemedicine are calculated using registration data and interviews with clinicians and finance department staff. To overcome missing data, additional national databases will be linked to the HF registry if feasible. Heterogeneity of the effects of offering telemedicine compared with not offering on days alive without unplanned hospitalisations in 1 year is assessed across predefined patient characteristics using exploratory stratified analyses. The effects of telemedicine components are assessed by fitting separate models for component contrasts. ETHICS AND DISSEMINATION: The study has been approved by the Medical Ethics Committee 2021 of the University Medical Center Utrecht (the Netherlands). Results will be published in peer-reviewed journals and presented at (inter)national conferences. Effective telemedicine scenarios will be proposed among hospitals throughout the country and abroad, if applicable and feasible. TRIAL REGISTRATION NUMBER: NCT05654961.

Increased endorsement of TRIPOD and other reporting guidelines by high impact factor journals: survey of instructions to authors.

OBJECTIVES: To assess the endorsement of reporting guidelines by high impact factor journals over the period 2017-2022, with a specific focus on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. STUDY DESIGN AND SETTING: We searched the online 'instructions to authors' of high impact factor medical journals in February 2017 and in January 2022 for any reference to reporting guidelines and TRIPOD in particular. RESULTS: In 2017, 205 out of 337 (61%) journals mentioned any reporting guideline in their instructions to authors and in 2022 this increased to 245 (73%) journals. A reference to TRIPOD was provided by 27 (8%) journals in 2017 and 67 (20%) in 2022. Of those journals mentioning TRIPOD in 2022, 22% provided a link to the TRIPOD website and 60% linked to TRIPOD information on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network website. Twenty-five percent of the journals required adherence to TRIPOD. CONCLUSION: About three-quarters of high-impact medical journals endorse the use of reporting guidelines and 20% endorse TRIPOD. Transparent reporting is important in enhancing the usefulness of health research and endorsement by journals plays a critical role in this.

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed.

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement. STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs. RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD. CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.

Prognosis and prediction of antibiotic benefit in adults with clinically diagnosed acute rhinosinusitis: an individual participant data meta-analysis.

BACKGROUND: A previous individual participant data meta-analysis (IPD-MA) of antibiotics for adults with clinically diagnosed acute rhinosinusitis (ARS) showed a marginal overall effect of antibiotics, but was unable to identify patients that are most likely to benefit from antibiotics when applying conventional (i.e. univariable or one-variable-at-a-time) subgroup analysis. We updated the systematic review and investigated whether multivariable prediction of patient-level prognosis and antibiotic treatment effect may lead to more tailored treatment assignment in adults presenting to primary care with ARS. METHODS: An IPD-MA of nine double-blind placebo-controlled trials of antibiotic treatment (n=2539) was conducted, with the probability of being cured at 8-15 days as the primary outcome. A logistic mixed effects model was developed to predict the probability of being cured based on demographic characteristics, signs and symptoms, and antibiotic treatment assignment. Predictive performance was quantified based on internal-external cross-validation in terms of calibration and discrimination performance, overall model fit, and the accuracy of individual predictions. RESULTS: Results indicate that the prognosis with respect to risk of cure could not be reliably predicted (c-statistic 0.58 and Brier score 0.24). Similarly, patient-level treatment effect predictions did not reliably distinguish between those that did and did not benefit from antibiotics (c-for-benefit 0.50). CONCLUSIONS: In conclusion, multivariable prediction based on patient demographics and common signs and symptoms did not reliably predict the patient-level probability of cure and antibiotic effect in this IPD-MA. Therefore, these characteristics cannot be expected to reliably distinguish those that do and do not benefit from antibiotics in adults presenting to primary care with ARS.

Missing values and inconclusive results in diagnostic studies - A scoping review of methods.

Most diagnostic studies exclude missing values and inconclusive results from the analysis or apply simple methods resulting in biased accuracy estimates. This may be due to the lack of availability or awareness of appropriate methods. This scoping review aimed to provide an overview of strategies to handle missing values and inconclusive results in the reference standard or index test in diagnostic accuracy studies. Conducting a systematic literature search in MEDLINE, Cochrane Library, and Web of Science, we could identify many articles proposing methods for addressing missing values in the reference standard. There are also several articles describing methods regarding missing values or inconclusive results in the index test. The latter encompass imputation, frequentist and Bayesian likelihood, model-based, and latent class methods. While methods for missing values in the reference standard are regularly applied in practice, this is not true for methods addressing missing values and inconclusive results in the index test. Our comprehensive overview and description of available methods may raise further awareness of these methods and will enhance their application. Future research is needed to compare the performance of these methods under different conditions to give valid and robust recommendations for their usage in various diagnostic accuracy research scenarios.

Cohort profile of PLUTO: a perioperative biobank focusing on prediction and early diagnosis of postoperative complications.

PURPOSE: Although elective surgery is generally safe, some procedures remain associated with an increased risk of complications. Improved preoperative risk stratification and earlier recognition of these complications may ameliorate postoperative recovery and improve long-term outcomes. The perioperative longitudinal study of complications and long-term outcomes (PLUTO) cohort aims to establish a comprehensive biorepository that will facilitate research in this field. In this profile paper, we will discuss its design rationale and opportunities for future studies. PARTICIPANTS: Patients undergoing elective intermediate to high-risk non-cardiac surgery are eligible for enrolment. For the first seven postoperative days, participants are subjected to daily bedside visits by dedicated observers, who adjudicate clinical events and perform non-invasive physiological measurements (including handheld spirometry and single-channel electroencephalography). Blood samples and microbiome specimens are collected at preselected time points. Primary study outcomes are the postoperative occurrence of nosocomial infections, major adverse cardiac events, pulmonary complications, acute kidney injury and delirium/acute encephalopathy. Secondary outcomes include mortality and quality of life, as well as the long-term occurrence of psychopathology, cognitive dysfunction and chronic pain. FINDINGS TO DATE: Enrolment of the first participant occurred early 2020. During the inception phase of the project (first 2 years), 431 patients were eligible of whom 297 patients consented to participate (69%). Observed event rate was 42% overall, with the most frequent complication being infection. FUTURE PLANS: The main purpose of the PLUTO biorepository is to provide a framework for research in the field of perioperative medicine and anaesthesiology, by storing high-quality clinical data and biomaterials for future studies. In addition, PLUTO aims to establish a logistical platform for conducting embedded clinical trials. TRIAL REGISTRATION NUMBER: NCT05331118.

Minimal reporting improvement after peer review in reports of COVID-19 prediction models: systematic review.

OBJECTIVES: To assess improvement in the completeness of reporting coronavirus (COVID-19) prediction models after the peer review process. STUDY DESIGN AND SETTING: Studies included in a living systematic review of COVID-19 prediction models, with both preprint and peer-reviewed published versions available, were assessed. The primary outcome was the change in percentage adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines between pre-print and published manuscripts. RESULTS: Nineteen studies were identified including seven (37%) model development studies, two external validations of existing models (11%), and 10 (53%) papers reporting on both development and external validation of the same model. Median percentage adherence among preprint versions was 33% (min-max: 10 to 68%). The percentage adherence of TRIPOD components increased from preprint to publication in 11/19 studies (58%), with adherence unchanged in the remaining eight studies. The median change in adherence was just 3 percentage points (pp, min-max: 0-14 pp) across all studies. No association was observed between the change in percentage adherence and preprint score, journal impact factor, or time between journal submission and acceptance. CONCLUSIONS: The preprint reporting quality of COVID-19 prediction modeling studies is poor and did not improve much after peer review, suggesting peer review had a trivial effect on the completeness of reporting during the pandemic.

The relation between prediction model performance measures and patient selection outcomes for proton therapy in head and neck cancer.

BACKGROUND: Normal-tissue complication probability (NTCP) models predict complication risk in patients receiving radiotherapy, considering radiation dose to healthy tissues, and are used to select patients for proton therapy, based on their expected reduction in risk after proton therapy versus photon radiotherapy (ΔNTCP). Recommended model evaluation measures include area under the receiver operating characteristic curve (AUC), overall calibration (CITL), and calibration slope (CS), whose precise relation to patient selection is still unclear. We investigated how each measure relates to patient selection outcomes. METHODS: The model validation and consequent patient selection process was simulated within empirical head and neck cancer patient data. By manipulating performance measures independently via model perturbations, the relation between model performance and patient selection was studied. RESULTS: Small reductions in AUC (-0.02) yielded mean changes in ΔNTCP between 0.9-3.2 %, and single-model patient selection differences between 2-19 %. Deviations (-0.2 or +0.2) in CITL or CS yielded mean changes in ΔNTCP between 0.3-1.4 %, and single-model patient selection differences between 1-10 %. CONCLUSIONS: Each measure independently impacts ΔNTCP and patient selection and should thus be assessed in a representative sufficiently large external sample. Our suggested practical model selection approach is considering the model with the highest AUC, and recalibrating it if needed.

Diagnostic prediction models for CT-confirmed and bacterial rhinosinusitis in primary care: individual participant data meta-analysis.

BACKGROUND: Antibiotics are overused in patients with acute rhinosinusitis (ARS) as it is difficult to identify those who benefit from antibiotic treatment. AIM: To develop prediction models for computed tomography (CT)-confirmed ARS and culture-confirmed acute bacterial rhinosinusitis (ABRS) in adults presenting to primary care with symptoms suggestive of ARS. DESIGN AND SETTING: This was a systematic review and individual participant data meta-analysis. METHOD: CT-confirmed ARS was defined as the presence of fluid level or total opacification in any maxillary sinuses, whereas culture-confirmed ABRS was defined by culture of fluid from antral puncture. Prediction models were derived using logistic regression modelling. RESULTS: Among 426 patients from three studies, 140 patients (32.9%) had CT-confirmed ARS. A model consisting of seven variables: previous diagnosis of ARS, preceding upper respiratory tract infection, anosmia, double sickening, purulent nasal discharge on examination, need for antibiotics as judged by a physician, and C-reactive protein (CRP) showed an optimism-corrected c-statistic of 0.73 (95% confidence interval [CI] = 0.69 to 0.78) and a calibration slope of 0.99 (95% CI = 0.72 to 1.19). Among 225 patients from two studies, 68 patients (30.2%) had culture-confirmed ABRS. A model consisting of three variables: pain in teeth, purulent nasal discharge, and CRP showed an optimism-corrected c-statistic of 0.70 (95% CI = 0.63 to 0.77) and a calibration slope of 1.00 (95% CI = 0.66 to 1.52). Clinical utility analysis showed that both models could be useful to rule out the target condition. CONCLUSION: Simple prediction models for CT-confirmed ARS and culture-confirmed ABRS can be useful to safely reduce antibiotic use in adults with ARS in high-prescribing countries.

Performance of binary prediction models in high-correlation low-dimensional settings: a comparison of methods.

BACKGROUND: Clinical prediction models are developed widely across medical disciplines. When predictors in such models are highly collinear, unexpected or spurious predictor-outcome associations may occur, thereby potentially reducing face-validity of the prediction model. Collinearity can be dealt with by exclusion of collinear predictors, but when there is no a priori motivation (besides collinearity) to include or exclude specific predictors, such an approach is arbitrary and possibly inappropriate. METHODS: We compare different methods to address collinearity, including shrinkage, dimensionality reduction, and constrained optimization. The effectiveness of these methods is illustrated via simulations. RESULTS: In the conducted simulations, no effect of collinearity was observed on predictive outcomes (AUC, R2, Intercept, Slope) across methods. However, a negative effect of collinearity on the stability of predictor selection was found, affecting all compared methods, but in particular methods that perform strong predictor selection (e.g., Lasso). Methods for which the included set of predictors remained most stable under increased collinearity were Ridge, PCLR, LAELR, and Dropout. CONCLUSIONS: Based on the results, we would recommend refraining from data-driven predictor selection approaches in the presence of high collinearity, because of the increased instability of predictor selection, even in relatively high events-per-variable settings. The selection of certain predictors over others may disproportionally give the impression that included predictors have a stronger association with the outcome than excluded predictors.

Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review.

While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.

Accuracy of approximations to recover incompletely reported logistic regression models depended on other available information.

OBJECTIVE: To provide approximations to recover the full regression equation across different scenarios of incompletely reported prediction models that were developed from binary logistic regression. STUDY DESIGN AND SETTING: In a case study, we considered four common scenarios and illustrated their corresponding approximations: (A) Missing: the intercept, Available: the regression coefficients of predictors, overall frequency of the outcome and descriptive statistics of the predictors; (B) Missing: regression coefficients and the intercept, Available: a simplified score; (C) Missing: regression coefficients and the intercept, Available: a nomogram; (D) Missing: regression coefficients and the intercept, Available: a web calculator. RESULTS: In the scenario A, a simplified approach based on the predicted probability corresponding to the average linear predictor was inaccurate. An approximation based on the overall outcome frequency and an approximation of the linear predictor distribution was more accurate, however, the appropriateness of the underlying assumptions cannot be verified in practice. In the scenario B, the recovered equation was inaccurate due to rounding and categorization of risk scores. In the scenarios C and D, the full regression equation could be recovered with minimal error. CONCLUSION: The accuracy of the approximations in recovering the regression equation varied depending on the available information.

Remote Hospital Care for Recovering COVID-19 Patients Using Telemedicine: A Randomised Controlled Trial.

BACKGROUND: To ensure availability of hospital beds and improve COVID-19 patients' well-being during the ongoing pandemic, hospital care could be offered at home. Retrospective studies show promising results of deploying remote hospital care to reduce the number of days spent in the hospital, but the beneficial effect has yet to be established. METHODS: We conducted a single centre, randomised trial from January to June 2021, including hospitalised COVID-19 patients who were in the recovery stage of the disease. Hospital care for the intervention group was transitioned to the patient's home, including oxygen therapy, medication and remote monitoring. The control group received in-hospital care as usual. The primary endpoint was the number of hospital-free days during the 30 days following randomisation. Secondary endpoints included health care consumption during the follow-up period and mortality. RESULTS: A total of 62 patients were randomised (31 control, 31 intervention). The mean difference in hospital-free days was 1.7 (26.7 control vs. 28.4 intervention, 95% CI of difference -0.5 to 4.2, p = 0.112). In the intervention group, the index hospital length of stay was 1.6 days shorter (95% CI -2.4 to -0.8, p < 0.001), but the total duration of care under hospital responsibility was 4.1 days longer (95% CI 0.5 to 7.7, p = 0.028). CONCLUSION: Remote hospital care for recovering COVID-19 patients is feasible. However, we could not demonstrate an increase in hospital-free days in the 30 days following randomisation. Optimising the intervention, timing, and identification of patients who will benefit most from remote hospital care could improve the impact of this intervention.

QUADAS-C: A Tool for Assessing Risk of Bias in Comparative Diagnostic Accuracy Studies.

Comparative diagnostic test accuracy studies assess and compare the accuracy of 2 or more tests in the same study. Although these studies have the potential to yield reliable evidence regarding comparative accuracy, shortcomings in the design, conduct, and analysis may bias their results. The currently recommended quality assessment tool for diagnostic test accuracy studies, QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies-2), is not designed for the assessment of test comparisons. The QUADAS-C (Quality Assessment of Diagnostic Accuracy Studies-Comparative) tool was developed as an extension of QUADAS-2 to assess the risk of bias in comparative diagnostic test accuracy studies. Through a 4-round Delphi study involving 24 international experts in test evaluation and a face-to-face consensus meeting, an initial version of the tool was developed that was revised and finalized following a pilot study among potential users. The QUADAS-C tool retains the same 4-domain structure of QUADAS-2 (Patient Selection, Index Test, Reference Standard, and Flow and Timing) and comprises additional questions to each QUADAS-2 domain. A risk-of-bias judgment for comparative accuracy requires a risk-of-bias judgment for the accuracy of each test (resulting from QUADAS-2) and additional criteria specific to test comparisons. Examples of such additional criteria include whether participants either received all index tests or were randomly assigned to index tests, and whether index tests were interpreted with blinding to the results of other index tests. The QUADAS-C tool will be useful for systematic reviews of diagnostic test accuracy addressing comparative questions. Furthermore, researchers may use this tool to identify and avoid risk of bias when designing a comparative diagnostic test accuracy study.